A #hashtagtokenizer for Social Media Messages

In social media, mainly due to length constraints, users write succinct messages and use hashtags to refer to entities, events, sentiments or ideas. Hashtags carry a lot of content that can help in many tasks and applications involving text processing such as sentiment analysis, named entity recognition and information extraction. However, identifying the individual words of a hashtag is not trivial because the traditional POS taggers typically consider it as a single token, despite the fact that it might contain multiple words, e.g. #fergusondecision, #imcharliehebdo. In this work, we propose a generic model for hashtagtokenisation that aims to split up one hashtag into several tokens corresponding to each individual word contained in it (e.g. “#imcharliehebdo” would become four tokens, “#”, “i”, “am” and “Charlie Hebdo”). Our hashtagtokenizer is based on a machine learning segmentation method for Chinese language and makes also use of Wikipedia as encyclopedic knowledge base. We have evaluated the inference power of our approach by comparing the tokens produced by our approach to those produced by human taggers. The results demonstrated the good accuracy and applicability of the proposed model for general-purpose applications.